Self-supervised pre-training of a speech foundation model, followed by supervised fine-tuning, has shown impressive quality improvements on automatic speech recognition (ASR) tasks. Fine-tuning separate foundation models for many downstream tasks are expensive since the foundation model is usually very big. Parameter-efficient fine-tuning methods (e.g. adapter, sparse update methods) offer an alternative paradigm where a small set of parameters are updated to adapt the foundation model to new tasks. However, these methods still suffer from a high computational memory cost and slow training speed because they require backpropagation through the entire neural network at each step. In the paper, we analyze the performance of features at different layers of a foundation model on the speech recognition task and propose a novel hierarchical feature fusion method for resource-efficient transfer learning from speech foundation models. Experimental results show that the proposed method can achieve better performance on speech recognition task than existing algorithms with fewer number of trainable parameters, less computational memory cost and faster training speed. After combining with Adapters at all layers, the proposed method can achieve the same performance as fine-tuning the whole model with $97\%$ fewer trainable encoder parameters and $53\%$ faster training speed.

在非洲使用的2,000多种语言几乎都没有广泛可用的自动语音识别系统，并且所需的数据也仅适用于几种语言。我们已经尝试了两种技术，这些技术可能为非洲语言提供大型词汇识别的途径：多语言建模和自我监督学习。我们收集了可用的开源数据并收集了15种语言的数据，并使用这些技术训练了实验模型。我们的结果表明，汇总多语言端到端模型中可用的少量数据，并预先培训无监督的数据可以帮助提高许多非洲语言的语音识别质量。

在移动设备上的语音模型（在设备个性化）上的个性化是一个活跃的研究领域，但是通常，移动设备比配对的音频文本数据具有更多的仅文本数据。我们探索培训有关仅文本数据的个性化语言模型，该模型在推理期间用于提高该用户的语音识别性能。我们在一个用户群体的Librispeech语料库上进行了实验，并为Gutenberg Project的每个用户提供了个性化的文本数据。我们发布此特定于用户的LibrisPeech（UserLibri）数据集，以帮助未来的个性化研究。LibrisPeech音频转录对分为来自测试清洁数据集的55个用户，另外有52位用户。我们能够降低流媒体和非启动模型中的两个集合中每个用户的平均单词错误率，包括在流式传输时为更难的测试用户组的2.5改进。

最先进的自动语音识别（ASR）系统经过数以万计的标记语音数据训练。人类转录很昂贵且耗时。诸如转录的质量和一致性之类的因素可以极大地影响使用这些数据训练的ASR模型的性能。在本文中，我们表明我们可以通过利用最近的自学和半监督学习技术来培训强大的教师模型来生产高质量的伪标签。具体来说，我们仅使用（无监督/监督培训）和迭代嘈杂的学生教师培训来培训6亿个参数双向教师模型。该模型在语音搜索任务上达到了4.0％的单词错误率（WER），比基线相对好11.1％。我们进一步表明，通过使用这种强大的教师模型来生成用于训练的高质量伪标签，与使用人类标签相比，流媒体模型可以实现13.6％的相对减少（5.9％至5.1％）。

自我监督的培训表明预先训练模型的有希望的收益，并促进了对语音识别的下游尖端，如多语言ASR。大多数现有方法采用一个2阶段方案，其中自我监督损失在第一个预先预订阶段进行了优化，并在第二阶段的标准监督的FINETUNING恢复。在本文中，我们提出了一部结束（E2E）联合无监督和监督培训（Just）方法，以将监督的RNN-T损失和自我监督的对比和屏蔽语言建模（MLM）损失结合起来。我们在公共数据集多语言LibrisPeech（MLS）上验证其性能，其中包括8种语言，非常不平衡。在MLS上，我们探索（1）刚从划痕训练，（2）刚从佩戴检查站训练。实验表明，只需始终如一地胜过其他现有的最先进的方法，并通过显着的保证金击败单声道基线，展示了在多语言ASR中处理低资源语言的能力。我们的平均WER所有语言都优于平均单声道基线33.3％，最先进的2级XLSR达到32％。在低资源语言如波兰语，我们的WER不到一半的单机基线，甚至击败了使用外部监管的监督转移学习方法。

我们总结了使用巨大的自动语音识别（ASR）模型的大量努力的结果，该模型使用包含大约一百万小时音频的大型，多样的未标记数据集进行了预训练。我们发现，即使对于拥有数万个小时的标记数据的非常大的任务，预训练，自我培训和扩大模型大小的组合也大大提高了数据效率。特别是，在具有34K小时标记数据的ASR任务上，通过微调80亿个参数预先训练的构象异构体模型，我们可以匹配最先进的（SOTA）性能（SOTA）的性能，只有3％的培训数据和通过完整的训练集可以显着改善SOTA。我们还报告了从使用大型预训练和自我训练的模型来完成一系列下游任务所获得的普遍利益，这些任务涵盖了广泛的语音域，并涵盖了多个数据集大小的大小，包括在许多人中获得SOTA性能公共基准。此外，我们利用预先训练的网络的学会表示，在非ASR任务上实现SOTA结果。

Existing 3D-aware image synthesis approaches mainly focus on generating a single canonical object and show limited capacity in composing a complex scene containing a variety of objects. This work presents DisCoScene: a 3Daware generative model for high-quality and controllable scene synthesis. The key ingredient of our method is a very abstract object-level representation (i.e., 3D bounding boxes without semantic annotation) as the scene layout prior, which is simple to obtain, general to describe various scene contents, and yet informative to disentangle objects and background. Moreover, it serves as an intuitive user control for scene editing. Based on such a prior, the proposed model spatially disentangles the whole scene into object-centric generative radiance fields by learning on only 2D images with the global-local discrimination. Our model obtains the generation fidelity and editing flexibility of individual objects while being able to efficiently compose objects and the background into a complete scene. We demonstrate state-of-the-art performance on many scene datasets, including the challenging Waymo outdoor dataset. Project page: https://snap-research.github.io/discoscene/

Modern autonomous driving system is characterized as modular tasks in sequential order, i.e., perception, prediction and planning. As sensors and hardware get improved, there is trending popularity to devise a system that can perform a wide diversity of tasks to fulfill higher-level intelligence. Contemporary approaches resort to either deploying standalone models for individual tasks, or designing a multi-task paradigm with separate heads. These might suffer from accumulative error or negative transfer effect. Instead, we argue that a favorable algorithm framework should be devised and optimized in pursuit of the ultimate goal, i.e. planning of the self-driving-car. Oriented at this goal, we revisit the key components within perception and prediction. We analyze each module and prioritize the tasks hierarchically, such that all these tasks contribute to planning (the goal). To this end, we introduce Unified Autonomous Driving (UniAD), the first comprehensive framework up-to-date that incorporates full-stack driving tasks in one network. It is exquisitely devised to leverage advantages of each module, and provide complementary feature abstractions for agent interaction from a global perspective. Tasks are communicated with unified query design to facilitate each other toward planning. We instantiate UniAD on the challenging nuScenes benchmark. With extensive ablations, the effectiveness of using such a philosophy is proven to surpass previous state-of-the-arts by a large margin in all aspects. The full suite of codebase and models would be available to facilitate future research in the community.

As a powerful representation of 3D scenes, the neural radiance field (NeRF) enables high-quality novel view synthesis from multi-view images. Stylizing NeRF, however, remains challenging, especially on simulating a text-guided style with both the appearance and the geometry altered simultaneously. In this paper, we present NeRF-Art, a text-guided NeRF stylization approach that manipulates the style of a pre-trained NeRF model with a simple text prompt. Unlike previous approaches that either lack sufficient geometry deformations and texture details or require meshes to guide the stylization, our method can shift a 3D scene to the target style characterized by desired geometry and appearance variations without any mesh guidance. This is achieved by introducing a novel global-local contrastive learning strategy, combined with the directional constraint to simultaneously control both the trajectory and the strength of the target style. Moreover, we adopt a weight regularization method to effectively suppress cloudy artifacts and geometry noises which arise easily when the density field is transformed during geometry stylization. Through extensive experiments on various styles, we demonstrate that our method is effective and robust regarding both single-view stylization quality and cross-view consistency. The code and more results can be found in our project page: https://cassiepython.github.io/nerfart/.

Software engineers working with the same programming language (PL) may speak different natural languages (NLs) and vice versa, erecting huge barriers to communication and working efficiency. Recent studies have demonstrated the effectiveness of generative pre-training in computer programs, yet they are always English-centric. In this work, we step towards bridging the gap between multilingual NLs and multilingual PLs for large language models (LLMs). We release ERNIE-Code, a unified pre-trained language model for 116 NLs and 6 PLs. We employ two methods for universal cross-lingual pre-training: span-corruption language modeling that learns patterns from monolingual NL or PL; and pivot-based translation language modeling that relies on parallel data of many NLs and PLs. Extensive results show that ERNIE-Code outperforms previous multilingual LLMs for PL or NL across a wide range of end tasks of code intelligence, including multilingual code-to-text, text-to-code, code-to-code, and text-to-text generation. We further show its advantage of zero-shot prompting on multilingual code summarization and text-to-text translation. We will make our code and pre-trained models publicly available.